Skip to content

Add Directory STructure for BackendBench #90

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 13 commits into
base: main
Choose a base branch
from
Draft

Conversation

msaroufim
Copy link
Member

@msaroufim msaroufim commented Aug 18, 2025

Alright lots of moving parts here and apologies for the lengthy PR but I'm opening this primarily to communicate with @Laurawly and @shaahins. This PR is in the same vein as #79

The goal here is to have high confidence that DirectoryBackend is correct and that means

  1. We aren't missing any torchbench operators (we will expand to others after launch)
  2. For any ops that we do implement we're confident they get overridden by our custom implementations
  3. We don't stretch to new features if we have low confidence they're working

To ensure we're not missing any torchbench operators I have two scripts that generate a fully functional DirectoryBackend with a root generated_kernels/ and each op name underneath it as a seperate folder. The mapping between a PyTorch operator and folder name can sometimes be tricky. For example what do you expect relu.default to default to and so if you're unsure you can generate them with the full result here. In addition as I went through this exercise some ops didn't seem to have much documentation about them despite being in torchbench, so I made a list of ops with no docstrings here

For any ops that we do implement we're confident they get overridden: The problem here is if we generate a correct operator and rely on logging statement we might still be loading the eager function since a correct function would be hard to distinguish. Instead what I do is purposefully load kernels that are incorrect with watermarked values of 42 that way we know the wrong kernel was loaded by us

Stretch of new features: one of the pitches of BackendBench was that this is a package you could install, I'd rather we tone down that pitch until we figure out details with the IntraKernel dispatcher so I'm deleting most of our code in __init__py

@msaroufim msaroufim marked this pull request as draft August 18, 2025 17:21
@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant